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ABSTRACT 
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Software, systems and methods for managing a distributed 
network environment including a plurality of computers 
interconnected by a network link, where at least some of the 
computers include a layered communications protocol stack 
for providing a data interface between an application pro- 
gram and the network link, the communications stack hav- 
ing a transport protocol layer for providing an end-to-end 
communications connection. The invention includes a con- 
trol module and a plurality of agent modules, each agent 
being associated with one of the computers and adapted to 
dynamically monitor the associated computer at a data 
transmission point between an application program running 
on the computer and the transport protocol layer and repeat- 
edly communicate with the control module in order to effect 
management of the distributed network system. The 
invented software, systems and methods may also include a 
messaging feature for providing users, IT personnel, or 
various management systems with informative messages 
concerning network conditions and network resources. 

7 Claims, 11 Drawing Sheets 
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SOFTWARE, SYSTEMS AND METHODS FOR 
MANAGING A DISTRIBUTED NETWORK 

FIELD OF THE INVENTION 

The present invention relates generally to distributed 
network systems, and more particularly to software, systems 
and methods for managing resources of a distributed net- 
work. 

BACKGROUND 

Both public and private networks have shifted toward a 
predominantly distributed computing model, and have 
grown steadily in size, power and complexity. This growth 
has been accompanied by a corresponding increase in 
demands placed on information technology to increase 
enterprise-level productivity, operations and customer/user 
support. To achieve interoperability in increasingly complex 
network systems, TCP/IP and other standardized communi- 
cation protocols have been aggressively deployed. Although 
many of these protocols have been effective at achieving 
interoperability, their widespread deployment has not been 
accompanied by a correspondingly aggressive development 
of management solutions for networks using these protocols. 

Indeed, conventional computer networks provide little in 
the way of solutions for managing network resources, and 
instead typically provide what is known as "best efforts"- 
service to all network traffic. Best efforts is the default 
behavior of TCP/IP networks, in which network nodes 
simply drop packets mdiscriminately when faced with 
excessive network congestion. With best efforts service, no 
mechanism is provided to avoid the congestion that leads to 
dropped packets, and network traffic is not categorized to 
ensure reliable delivery of more important data. Also, users 
are not provided with information about network conditions 
or underperforming resources. This lack of management 
frequently results in repeated, unsuccessful network 
requests, user frustration and diminished productivity. 

Problems associated with managing network resources 
are intensified by the dramatic increase in the demand for 
these resources. New applications for use in distributed 
networking environments are being developed at a rapid 
pace. These applications have widely varying performance 
requirements. Multimedia applications, for example, have a 
very high sensitivity to jitter, loss and delay. By contrast, 
other types of applications can tolerate significant lapses in 
network performance. Many applications, particularly con- 
tinuous media applications, have very high bandwidth 
requirements, while others have bandwidth requirements 
that are comparatively modest. A further problem is that 
many bandwidth -intensive applications are used for recre- 
ation or other low priority tasks. 

In the absence of effective management tools, the result of 
this increased and varied competition for network resources 
is congestion, application unpredictability, user frustration 
and loss of productivity. When networks are unable to 
distinguish unimportant tasks or requests from those that are 
mission critical, network resources are often used in ways 
that are inconsistent with business objectives. Bandwidth 
may be wasted or consumed by low priority tasks. Custom- 
ers may experience unsatisfactory network performance as a 
result of internal users placing a high load on the network. 

Various solutions have been employed, with limited 
success, to address these network management problems. 
For example, to alleviate congestion, network managers 
often add more bandwidth to congested links. This solution 
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is expensive and can be temporary — network usage tends to 
shift and grow such that the provisioned link soon becomes 
congested again. This often happens where the underlying 
cause of the congestion is not addressed. Usually, it is 

5 desirable to intelligently manage existing resources, as 
opposed to "over-provisioning," i.e. simply providing more 
resources to reduce scarcity. 

A broad, conceptual class of management solutions may 
be thought of as attempts to increase "awareness" in a 

10 distributed networking environment. The concept is that 
where the network is more aware of applications or other 
tasks running on networked devices, and vice versa, then 
steps can be taken to make more efficient use of network 
resources. For example, if network management software 
becomes aware that a particular user is running a low 

15 priority application, then the software could block or limit 
that user's access to network resources. If management 
software becomes aware that the network population at a 
given instance includes a high percentage of outside 
customers, bandwidth preferences and priorities could be 

20 modified to ensure that the customers had a positive expe- 
rience with the network. In the abstract, increasing applica- 
tion and network awareness is a desirable goal, however 
application vendors largely ignore these considerations and 
tend to focus not on network infrastructure, but rather on 

25 enhancing application functionality. 

Quality of service ("QoS") and policy -based management 
techniques represent efforts to bridge the gap between 
networks, applications and users in order to more efficiently 
manage the use of network resources. QoS is a term refer- 

30 ring to techniques which allow network- aware applications 
to request and receive a predictable level of service in terms 
of performance specifications such as bandwidth, jitter, 
delay and loss. Known QoS methods include disallowing 
certain types of packets, slowing transmission rates, estab- 

35 lishing distinct classes of services for certain types of 
packets, marking packets with a priority value, and various 
queuing methods. In a distributed environment having 
scarce resources, QoS techniques necessarily introduce 
unfairness into the system by giving preferential treatment to 

40 certain network traffic. 

Policy -based network management uses policies, or rules, 
to define how network resources are to be used. In a broad 
sense, a policy includes a condition and an action. An 
example of a policy could be to block access or disallow 

45 packets (action) if the IP source address of the data is 
included on a list of disallowed addresses (condition). One 
use of policy-based network management techniques is to 
determine when and how the unfairness introduced by QoS 
methods should apply. 

so Policy-based management solutions typically require that 
network traffic be classified before it is acted upon. The 
classification process can occur at various levels of data 
abstraction, and may be described in terms of layered 
communication protocols that network devices use to com- 

55 municate across a network link. There are two protocol 
layering models which dominate the field. The first is the 
OSI reference model, depicted in FIG. 1. The layers of the 
OSI model are: application (layer 7), presentation (layer 6), 
session (layer 5), transport (layer 4), network (layer 3), data 

60 fink (layer 2) and physical (layer 1). The second major 
model forms the basis for the TCP/IP protocol suite. Its 
layers are application, transport, network, data link and 
hardware, as also depicted in FIG. 1. The TCP/IP layers 
correspond in function to the OSI layers, but without a 

65 presentation or session layer. In both models, data is pro- 
cessed and changes form as it is sequentially passed between 
the layers. 
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Known policy based management solutions and QoS 
methods typically classify data by monitoring data flows at 
the transport layer and below. For example, a common 
multi-parameter classifier is the well known "five-tuple" 
consisting of (IP source address, IP destination address, IP 
protocol, TCP/UDP source port and TCP/UDP destination 
port). These parameters are all obtained at the transport and 
network layers of the models. The large majority of existing 
policy-based, QoS solutions are implemented by monitoring 
and classifying network activity at these protocol layers. 
However, the higher the protocol layer, the more definitive 
and specific the available data and classifiers. Because 
conventional policy-based, QoS systems do not employ 
classifiers at higher than the transport layer, they cannot 
employ policy -based techniques or QoS methods using the 
richer and more detailed data available at the higher layers. 
The conventional systems are thus limited in their ability to 
make the network more application-aware and vice versa. 

In addition, the known systems for managing network 
resources do not effectively address the problem of band- 
width management. Bandwidth is often consumed by low 
priority tasks at the expense of business critical applications. 
In systems that do provide for priority based bandwidth 
allocations, the bandwidth allocations are static and are not 
adjusted dynamically in response to changing network con- 
ditions. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a conceptual depiction of the OSI and TCP/IP 
layered protocol models. 

FIG. 2 is a view of a distributed network system in which 
the software, systems and methods of the present invention 
may be deployed. 

FIG. 3 is a schematic view of a computing device that may 
be included with the distributed network system of FIG. 2. 

FIG. 4 is a block diagram view depicting the relationship 
between the distributed agent modules and control modules 
of the present invention, and depicting the relationship 
between the computing devices with which the agent and 
control modules are associated. 

FIG. 5 is a block diagram view depicting various com- 
ponents of the invented software, system and methods, 
including two agent modules, a control module and a 
configuration utility, the diagram also depicting the inter- 
connection of the components with layered communications 
protocol software and a network link. 

FIG. 6 is a block diagram depicting a configuration of an 
agent module according to the present invention with a 
computing device. 

FIG. 7 is a block diagram depicting another configuration 
of an agent module according to the present invention with 
a computing device. 

FIG. 8 is a block diagram depicting yet another configu- 
ration of an agent module according to the present invention 
with a computing device. 

FIG. 9 is a block diagram depicting various component 
parts of an embodiment of an agent module according to the 
present invention. 

FIG. 10 is a block diagram depicting various component 
parts of an embodiment of a control module according to the 
present invention. 

FIG. 11 A is a flowchart depicting a method according to 
the present invention for allocating bandwidth among a 
plurality of computers. 

FIG. 11 B is a flowchart depicting another method accord- 
ing to the present invention for allocating bandwidth among 
a plurality of computers. 
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FIG. 11C is a flowchart depicting yet another method 
according to the present invention for allocating bandwidth 
among a plurality of computers. 

FIG. 11D is a flowchart depicting yet another method 
5 according to the present invention for allocating bandwidth 
among a plurality of computers. 

FIG. 12 is a flowchart depicting a method according to the 
present invention for monitoring the status of network 
resources. 

FIG. 13 is a view of a main configuration screen of a 
configuration utility according to the present invention, 

FIG. 14 is a view of another configuration screen of the 
configuration utility. 
15 FIG. IS is a view of yet another configuration screen of 
the configuration utility. 

FIG. 16 is a view of yet another configuration screen of 
the configuration utility. 

20 DETAILED DESCRIPTION OF THE 

INVENTION 

The present invention provides a system and method for 
managing network resources in a distributed networking 
environment, such as distributed network 10 depicted in 

25 FIG. 2. The invented software, system and methods increase 
productivity and customer/user satisfaction, minimize frus- 
tration associated with using the network, and ultimately 
ensure that network resources are used in a way consistent 
with underlying business or other objectives. 

The invention includes two main software components, 
an agent and a control module, also referred to as a control 
point. The agents and control points are deployed throughout 
distributed network 10, and interact with each other to 

35 accomplish the above goals. A plurality of agents may be 
deployed to intelligently couple clients, servers and other 
computing devices to the underlying network. The deployed 
agents monitor, analyze and act upon network events relat- 
ing to the networked devices with which they are associated. 

40 The agents are centrally coordinated and/or controlled by 
one or more control points. The agents and control points 
interact to control and monitor network events, track opera- 
tional and congestion status of network resources, select 
optimum targets for network requests, dynamically manage 

45 bandwidth usage, and share information about network 
conditions with customers, users and IT personnel. 

As indicated, distributed network 10 may include a local 
network 12 and a plurality of remote networks 14 linked by 
a public network 16 such as the Internet. The local network 

50 and remote networks are connected to the public network 
with network infrastructure devices such as routers 18. 

Local network 12 will be seen to include servers 20 and 
client devices such as client computers 22 interconnected by 
network link 24. Additionally, local network 12 may include 

55 any number and variety oLdevices, including file servers, 
applications servers, mail servers, WWW servers, databases, 
client computers, remote access devices, storage devices, 
printers and network infrastructure devices such as routers, 
bridges, gateways, switches, hubs and repeaters. Remote 

60 networks 14 may similarly include any number and variety 
of networked devices. 

Indeed, virtually any type of computing device may be 
connected to the networks depicted in FIG. 2, including 
general purpose computers, laptop computers, handheld 

65 computers, wireless computing devices, mobile telephones, 
pagers, pervasive computing devices and various other spe- 
cialty devices. Typically, many of the connected devices are 
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general purpose computers which have at least some of the 
elements shown in FIG. 3, a block diagram depiction of a 
computer system 40. Computer system 40 includes a pro- 
cessor 42 that processes digital data. The processor may be 
a complex instruction set computing (CISC) 
microprocessor, a reduced instruction set computing (RISC) 
microprocessor, a very long instruction word (VUW) 
microprocessor, a processor implementing a combination of 
instruction sets, a microcontroller, or virtually any other 
processor/controller device. The processor may be a single 
device or a plurality of devices. 

Referring still to FIG. 3, it will be noted that processor 42 
is coupled to a bus 44 which transmits signals between the 
processor and other components in the computer system. 
Those skilled in the art will appreciate that the bus may be 
a single bus or a plurality of buses. A memory 46 is coupled 
to bus 44 and comprises a random access memory (RAM) 
device 47 (referred to as main memory) that stores infor- 
mation or other intermediate data during execution by 
processor 42. Memory 46 also includes a read only memory 
(ROM) and/or other static storage device 48 coupled to the 
bus that stores information and instructions for processor 42. 
A basic input/output system (BIOS) 49, containing the basic 
routines that help to transfer information between elements 
of the computer system, such as during start-up, is stored io 
ROM 48. A data storage device 50 also is coupled to bus 44 
and stores information and instructions. The data storage 
device may be a hard disk drive, a floppy disk drive, a 
CD-ROM device, a flash memory device or any other mass 
storage device. In the depicted computer system, a network 
interface 52 also is coupled to bus 44. The network interface 
operates to connect the computer system to a network (not 
shown). 

Computer system 40 may also include a display device 
controller 54 coupled to bus 44. The display device control- 
ler allows coupling of a display device to the computer 
system and operates to interface the display device to the 
computer system. The display device controller 54 may be, 
for example, a monochrome display adapter (MDA) card, a 
color graphics adapter (CGA) card, or other display device 
controller. The display device (not shown) may be a televi- 
sion set, a computer monitor, a flat panel display or other 
display device. The display device receives information and 
data from processor 42 through display device controller 54 
and displays the information and data to the user of com- 
puter system 40. 

An input device 56, including alphanumeric and other 
keys, typically is coupled to bus 44 for communicating 
information and command selections to processor 42. 
Alternatively, input device 56 is not directly coupled to bus 
44, but interfaces with the computer system via infra-red 
coded signals transmitted from the input device to an 
infra-red receiver in the computer system (not shown). The 
input device may also be a remote control unit having keys 
that select characters or command selections on the display 
device. 

The various computing devices coupled to the networks 
of FIG. 2 typically communicate with each other across 
network links using communications software employing 
various communications protocols. The communications 
software for each networked device typically consists of a 
number of protocol layers, through which data is sequen- 
tially transferred as it is exchanged across a network link 
between devices. FIG. 1 respectively depicts the OSI layered 
protocol model and a layered model based on the TCP/IP 
suite of protocols. These two models dominate the field of 
network communications software. As seen in the figure, the 
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OSI model has seven layers, including an application layer, 
a presentation layer, a session layer, a transport layer, a 
network layer, a data link layer and a physical layer. The 
TCP/IP-based model includes an application layer, a trans- 
5 port layer, a network layer, a data link layer and a physical 
layer. 

Each layer in the models plays a different role in network 
communications. Conceptually, all of the protocol layers lie 
in a data transmission path that is "between" an application 

1° program running on the particular networked device and the 
network link, with the application layer being closest to the 
application program. When data is transferred from an 
application program running on one computer across the 
network to an application program running on another 

15 computer, the data is transferred down through the protocol 
layers of the first computer, across the network link, and then 
up through the protocol layers on the second computer. 

In both of the depicted models, the application layer is 
responsible for interacting with an operating system of the 

20 networked device and for providing a window for applica- 
tion programs running on the device to access the network. 
The transport layer is responsible for providing reliable, 
end-to-end data transmission between two end points on a 
network, such as between a client device and a server 

25 computer, or between a web server and a DNS server. 
Depending on the particular transport protocol, transport 
functionality may be realized using either connection- 
oriented or connectionless data transfer. The network layer 
typically is not concerned with end-to-end delivery, but 

30 rather with forwarding and routing data to and from nodes 
between endpoints. The layers below the transport and 
network layers perform other functions, with the lowest 
levels addressing the physical and electrical issues of trans- 
mitting raw bits across a network link. 

The present invention is applicable to a wide variety of 
network environments employing communications proto- 
cols adhering to either of the layered models depicted in 
FIG. 1, or to any other layered model. Furthermore, the 

40 invention is applicable to any type of network topology, and 
to networks using both physical and wireless connections. 

The present invention provides software, systems and 
methods for managing the resources of an enterprise 
network, such as that depicted in FIG. 2. This is accom- 

45 plished using two main interacting software components, an 
agent and a control point, both of which are adapted to run 
on, or be associated with, computing devices such as the 
computing device described with reference to FIG. 3. As 
seen in FIG. 4, a plurality of agents 70 and one or more 

50 control points 72 may be deployed throughout distributed 
network 74 by loading the agent and control point software 
modules on networked computing devices such as clients 22 
and server 20. As will be discussed in detail, the agents and 
control points may be adapted and configured to enforce 

55 system policies; to monitor and analyze network events, and 
take appropriate action based on these events; to provide 
valuable information to users of the network; and ultimately 
to ensure that network resources are efficiently used in a 
manner consistent with underlying business or other goals. 

60 The invented software, systems and methods may be 
configured using a third software component, to be later 
discussed in more detail with reference to FIGS. 5 and 
13-16. Typically, this configuration utility is a platform- 
independent application that provides a graphical user inter- 

65 face for centrally managing configuration information for 
the control points and agents. In addition, the configuration 
utility may be adapted to communicate and interface with 
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other management systems, including management plat- 
forms supplied by other vendors. 

As indicated in FIG. 4, each control point 72 is typically 
associated with multiple agents 70, and the associated agents 
arc referred to as being within a domain 76 of the particular 
control point The control points coordinate and control the 
activity of the distributed agents within their domains. In 
addition, the control points monitor the status of network 
resources, and share this information with management and 
support systems and with the agents. 

Control points 72 and agents 70 may be flexibly deployed 
in a variety of configurations. For example, each agent may 
be associated with a primary control point and one or more 
backup control points that will assume primary control if 
necessary. Such a configuration is illustrated in FIG. 4, 
where control points 72 within the dashed lines function as 
primary connections, with the control point associated with 
server device 20 functioning as a backup connection for all 
of the depicted agents. In addition, the invented management 
solution may be deployed so that one control point coordi- 
nates and controls the activity of a single domain, or of 
multiple domains. Alternatively, one domain may be con- 
trolled and coordinated by the cooperative activity of mul- 
tiple control points. In addition, agents may be configured to 
have embedded control point functionality, and may there- 
fore operate without an associated control point entity. 

The agents monitor network resources and the activity of 
the device with which they are associated, and communicate 
this information to the control points. In response to moni- 
tored network conditions and data reported by agents, the 
control points alter the behavior of particular agents in order 
to provide the desired network services. The control points 
and agents may be loaded on a wide variety of devices, 
including general purpose computers, servers, routers, hubs, 
palm computers, pagers, cellular telephones, and virtually 
any other networked device having a processor and memory. 
Agents and control points may reside on separate devices, or 
simultaneously on the same device. 

FIG. 5 illustrates an example of the way in which the 
various components of the invented software, systems and 
methods may be physically interconnected with a network 
link 90. The components are all connected to network link 
90 by means of layered communications protocol software 
92. The components communicate with each other via the 
communications software and network link. As will be 
appreciated by those skilled in the art, network link 90 may 
be a physical or wireless connection, or a series of links 
including physical and wireless segments. More specifically, 
the depicted system includes an agent 70 associated with a 
client computing device 22, including an application pro- 
gram 98. Another agent is associated with server computing 
device 20. The agents monitor the activity of their associated 
computing devices and communicate with control point 72. 
Configuration utility 106 communicates with all of the other 
components, and with other management systems, to con- 
figure the operation of the various components and monitor 
the status of the network. 

The system policies that define how network resources are 
to be used may be centrally defined and tailored to most 
efficiently achieve underlying goals. Defined policies are 
accessed by the control points, which in turn communicate 
various elements and parameters associated with the policies 
to the agents within their domain. At a very basic level, a 
policy contains rules about how network resources are to be 
used, with the rules containing conditions and actions to be 
taken when the conditions are satisfied. The agents and 
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control points monitor the network and devices connected to 
the network to determine when various rules apply and 
whether the conditions accompanying those rules are satis- 
fied. Once the agents and/or control points determine that 

s action is required, they take the necessary action(s) to 
enforce the system policies. 

For example, successful businesses often strive to provide 
excellent customer services. This underlying business goal 
can be translated into many dififerent policies defining how 

10 network resources are to be used. One example of such a 
policy would be to prevent or limit access to non-business 
critical applications when performance of business critical 
applications is degraded beyond a threshold point. Another 
example would be to use QoS techniques to provide a 

15 guaranteed or high level of service to e-commerce applica- 
tions. Yet another example would be to dynamically increase 
the network bandwidth allocated to a networked computer 
whenever it is accessed by a customer. Also, bandwidth for 
various applications might be restricted during times when 

20 there is heavy use of network resources by customers. 

Control points 72 would access these policies and provide 
policy data to agents 70. Agents 70 and control points 72 
would communicate with each other and monitor the net- 
work to determine how many customers were accessing the 

25 network, what computers the customers) were accessing, 
and what applications were being accessed by the custom- 
ers. Once the triggering conditions were detected, the agents 
and control points would interact to re-allocate bandwidth, 
provide specified service levels, block or restrict various 

30 non-customer activities, etc. 

Another example of policy -based management would be 
to define an optimum specification of network resources or 
service levels for particular types of network tasks. The 

35 particular policies would direct the management entities to 
determine whether the particular task was permitted, and if 
permitted, the management entities would interact to ensure 
that the desired level of resources was provided to accom- 
plish the task. If the optimum resources were not available, 

4Q the applicable policies could further specify that the 
requested task be blocked, and that the requesting user be 
provided with an informative message detailing the reason 
why the request was denied. Alternatively, the policies could 
specify that the user be provided with various options, such 

45 as proceeding with the requested task, but with sub-optimal 
resources, or waiting to perform the task until a later time. 

For example, continuous media applications such as IP 
telephony have certain bandwidth requirements for optimum 
performance, and are particularly sensitive to network jitter 

50 and delay. Policies could be written to specify a desired level 
of service, including bandwidth requirements and threshold 
levels for jitter and delay, for client computers attempting to 
run IP telephony applications. The policies would further 
direct the agents and control modules to attempt to provide 

55 the specified level of service. Security checking could also 
be included to ensure that the particular user or client 
computer was permitted to run the application. In the event 
that the specified service level could not be provided, the 
requesting user could be provided with a message indicating 

60 that the resources for the request were not available. The 
user could also be offered various options, including pro- 
ceeding with a sub-optimal level of service, placing a 
conventional telephone call, waiting to perform the task 
until a later time, etc. 

65 The software, system and methods of the present inven- 
tion may be used to implement a wide variety of system 
policies. The policy rules and conditions may be based on 
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any number of parameters, including IP source address, IP 
destination address, source port, destination port, protocol, 
application identity, user identity, device identity, URL, 
available device bandwidth, application profile, server 
profile, gateway identity, router identity, time-of-day, net- 
work congestion, network load, network population, avail- 
able domain bandwidth and resource status, to name but a 
partial list. The actions taken when the policy conditions are 
satisfied can include blocking network access, adjusting 
service levels and/or bandwidth allocations for networked 
devices, blocking requests to particular URLs, diverting 
network requests away from overloaded or underperforming 
resources, redirecting network requests to alternate 
resources and gathering network statistics. 

Some of the parameters listed above may be thought of as 
"client parameters," because they are normally evaluated by 
an agent monitoring a single networked client device. These 
include IP source address, IP destination address, source 
port, destination port, protocol, application identity, user 
identity, available device bandwidth and URL. Other 
parameters, such as application profile, server profile, gate- 
way identity, router identity, time-of-day, network 
congestion, network load, network population, available 
domain bandwidth and resource status may be though of as 
"system parameters" because they pertain to shared 
resources, aggregate network conditions or require evalua- 
tion of data from multiple agent modules. Despite this, there 
is not a precise distinction between client parameters and 
system parameters. Certain parameters, such as time-of-day, 
may be considered either a client parameter or a system 
parameter, or both. 

Policy-based network management, QoS implementation, 
and the other functions of the agents and control points 
depend on obtaining real-time information about the net- 
work. As will be discussed, the present invention is an 
improvement over known policy-based QoS management 
solutions because of the enhanced ability to obtain detailed 
information about network conditions and the activity of 
networked devices. Many of the policy parameters and 
conditions discussed above are accessible in the present 
invention due to the particular way the agent modules are 
coupled to the communications software of their associated 
devices. Also, as the above examples suggest, managing 
bandwidth and ensuring its availability for core applications 
is an increasingly important consideration in managing 
networks. The present invention provides an improved sys- 
tem and method for dynamically allocating bandwidth and 
controlling bandwidth usage in response to changing net- 
work conditions. 

The ability of the present invention to flexibly deploy 
policy-based, QoS management solutions based on detailed 
information about network conditions has a number of 
significant benefits. These benefits include reducing frustra- 
tion associated with using the network, reducing help calls 
to IT personnel, increasing productivity, lowering business 
costs associated with managing and maintaining enterprise 
networks, and increased customer/user loyalty and satisfac- 
tion. Ultimately, the invented systems and methods ensure 
that network resources are used in a way that is consistent 
with underlying goals and objectives. 

The ability of the present invention to implement policy- 
based QoS between the application and transport layers has 
another advantage. This allows support for encryption and 
other security implementations carried out using Virtual 
Private Networking (VPN) or IPSec protocol. 

Referring now to FIGS, 6-9, the agent module will be 
more particularly described. The basic functions of the agent 



1,724 Bl 

10 

module are monitoring the status and activities of its asso- 
ciated client, server, pervasive computing device or other 
computing device, communicating this information to one or 
more control points, enforcing system policies under the 
s direction of the control points, and providing messages to 
network users and administrators concerning network con- 
ditions. FIGS. 6-8 are conceptual depictions of networked 
computing devices, and show how the agent software is 
associated with the networked devices relative to layered 
protocol software used by the devices for network commu- 
nication. 

As seen in FIG. 6, agent 70 is interposed between appli- 
cation program 122 and a communications protocol layer for 
providing end-to-end data transmission, such as transport 

15 layer 124 of communications protocol stack 92. Typically, 
the agent modules of the present invention are used with 
network devices that employ layered communications soft- 
ware adhering to either the OSI or TCP/IP-based protocol 
models. Thus, agent 70 is depicted as "interposed," i.e. in a 

20 data path, between an application program and a transport 
protocol layer. However, it will be appreciated by those 
skilled in the art that the invented agent may be used with 
protocol software not adhering to either the OSI or TCP/IP 
models, but that nonetheless includes a protocol layer pro- 

M viding transport functionality, i.e. providing for end-to-end 
data transmission. L 
Because of the depicted position within the data path, 
agent 70 is able to monitor network traffic and obtain 
information that is not available by hooking into transport 

30 layer 124 or the layers below the transport layer. At th 
higher layers, the available data is richer and more detailed. 
Hooking into the stack at higher layers allows the network 
to become more "application-aware" than is possible when 
monitoring occurs at the transport and lower layers. 

35 The agent modules may be interposed at a variety of 
points between application program 122 and transport layer 
124. Specifically, as shown in FIGS. 7 and 8, agent 70 may 
be associated with a client computer so that it is adjacent an 
application programming interface (API) adapted to provide 

40 a standardized interface for application program 122 to 
access a local operating system (not shown) and communi- 
cations stack 92. In FIG. 7, agent 70 is adjacent a winsock 
API 128 and interposed between application program 122 
and the winsock interface. FIG. 8 shows an alternate 

4 5 configuration, in which agent 70 is again adjacent the 
winsock interface, but the winsock interface is interposed 
between application program 122 and agent 70. With eithe 
configuration, the agent is interposed between the transport 
layer 124 of communications stack 92 and is adapted to 

50 direcdy monitor data received by or sent from the winsock 
interface. 

As shown in FIG. 8, agent 70 may be configured to hoo" 
into lower layers of communications stack 92. This allows 
the agent to accurately monitor network traffic volumes by 

55 providing a correction mechanism to account for data com- 
pression or encryption occurring at protocol layers below 
transport layer 124. For example, if compression or encryp- 
tion occurs within transport layer 124, monitoring at a point 
above the transport layer would yield an inaccurate measure 

60 of the network traffic associated with the computing device. 
Hooking into lower layers with agent 70 allows network 
traffic to be accurately measured in the event that 
compression, encryption or other data processing that quali- 
tatively or quantitatively affects network traffic occurs at 

65 lower protocol layers. 

An embodiment of the agent module is depicted in FIG. 
9. As shown, agent 70 may include a redirector module 130, 



02/10/2004, EAST version: 1.4.1 



US 6,671,724 Bl 

11 12 

a traffic control module 132, an administrator module 134, accumulates "credits" for intervals in which it does not 

a DNS module 136, a popapp module 138, a message broker release any waiting data. When enough credits are 

module 140, a system services module 142, and a popapp accumulated, the waiting message is released for forwarding 

144. Redirector module 130 intercepts winsock API calls to the network, 

made by applications running on networked devices such as 5 Similarly, to control the rate at which network traffic is 
the client computers depicted in FIGS. 2 and 3. Redirector received, traffic control module 132 may be configured to 
module 130 then hands these calls to one or more of the maintain a plurality of separate receive queues, such as 
other agent components for processing. As discussed with receive queues 132b. In addition to the methods discussed 
reference to FIGS. 6-8, redirector module is positioned to aD0VC ^ various other methods may be employed to control 
allow the agent to monitor data at a data transmission point 10 the rate at which network traffic is sent and received by the 
between an application program running on the device and queues. Also, the behavior of the transmit and receive 
the transport layer of the communications stack. Depending queues may be controlled through various methods to con- 
on the configuration of the agent and control point, the troj jitter, delay, loss and response time for network con- 
intercepted winsock calls may be rejected, changed, or nections. 

passed on by agent 70. is transmit ^ rece ive queues may also be configured 

Traffic control module 132 implements QoS and system t 0 detect network conditions such as congestion and slow 

policies and assists in monitoring network conditions. Traf- responding applications or servers. For example, for each 

fic control module 132 implements QoS methods by con- application, transmitted packets or other data units may be 

trolling the network traffic flow between applications run- timestamped when passed out of a transmit queue. When 

ning on the agent device and the network link. The traffic 20 corresponding packets are received for a particular 

flow is controlled to deliver a specified network service application, the receive and send times may be compared to 

level, which may include specifications of bandwidth, data detect network congestion and/or slow response times for 

throughput, jitter, delay and data loss. various target resources. This information may be reported 

To provide the specified network service level, traffic to the control points and shared with other agents within the 

control module 132 may maintain a queue or plurality of 25 domain. The response time and other performance infonna- 

queues. When data is sent from the client to the network, or tion obtained by comparing transmit and receive times may 

from the network to the client, redirector module 130 also be used to compile and maintain statistics regarding 

intercepts the data, and traffic module 132 places the indi- various network resources. 

vidual units of data in the appropriate queue. The control Using this detection and reporting mechanism, a controP 

points may be configured to periodically provide traffic point may be configured to reduce network loads by instruct- 

control commands, which may include the QoS parameters mg traffic control module 132 to close low priority sessions 

and service specifications discussed above. In response, an d block additional sessions whenever heavy network 

traffic control module 132 controls the passing of data into, congestion is reported by one of the agents. In conjunction, 

through or out of the queues in order to provide the specified ^ a s will be explained, popapp 138 module may provide a_ 

service level. message to the user explaining why sessions are being 

More specifically, the outgoing traffic rate may be con- closed. In addition to closing the existing sessions, the 

trolled using a plurality of priority-based transmission control point may be configured to instruct the agents to 

queues, such as transmission queues 132a. When an appli- block any further sessions. This action may also be accom- 

cation or process is invoked by a computing device with 4Q panied by a user message in response to attempts to launch 

which agent 70 is associated, a priority level is assigned to a new application or network process. When the network 

the application, based on centrally defined policies and load is reduced, the control point will send a message to the 

priority data supplied by the control point. Specifically, as agents allowing sessions. 

will be discussed, the control points maintain user profiles, In addition to identifying congestion and slow response 

applications profiles and network resource profiles. These 45 times, traffic control module 132 may be more generally 

profiles include priority data which is provided to the agents. configured to aid in identifying downed or under-performing 

Transmission queues 132a may be configured to release network resources. When a connection to a target resource 

data for transmission to the network at regular intervals. fails, traffic module 132 notifies popapp modules 138, which 

Using the parameters specified in traffic control commands in turn launches an executable to perform a root-cause 

issued by a control point, traffic module 132 calculates how 50 analysis of the problem. Agent 70 then provides the control 

much data can be released from the transmission queues in point with a message identifying the resource and its status, 

a particular interval. For example, if the specified average if possible. 

traffic rate is 100 KBps and the queue release interval is 1 In addition, when a connection fails, popapp module 138 

ms, then the total amount of data that the queues can release may be configured to provide a message to the user, includ- 

in a given interval is 100 bits. The relative priorities of the 55 ing an option to initiate an autoconnect routine targeting the 

queues containing data to be transmitted determine how unavailable resource. Enabling autoconnect causes the agent 

much of the allotment may be released by each individual to periodically retry the unavailable resource. This feature 

queue. For example, assuming there are only two queues, Ql may be disabled, if desired, to allow the control point to 

and Q2, that have data queued for transmission, Ql will be assume responsibility for determining when the resource 

permitted to transmit 66.66% of the overall allotted interval 60 becomes available again. As will be later discussed, the 

release if its priority is twice that of Q2. Q2 would only be invented system may be configured so that the control 

permitted to release 33.33% of the allotment. If their pri- modules assume responsibility for monitoring unavailable 

orities were equal, each queue would be permitted to release resources in order to minimize unnecessary network traffic. 

50% of the interval allotment for forwarding to the network As discussed below, various agent components also moni- 

un ^* 65 tor network conditions and resource usage for the purpose of 

If waiting data is packaged into units that are larger than compiling statistics. An additional function of traffic control 

the amount a given queue is permitted to release, the queue module 132 is to aid in performing these functions by 
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providing information to other agent components regarding 
accessed resources, including resource performance and 
frequency of access. 

As suggested in the above discussion of traffic control 
module 132, popapp module 138 stores and is responsible 
for launching a variety of small application modules such as 
application 144, known as popapps, to perform various 
operations and enhance the functioning of the invented 
system. Popapps detect and diagnose network conditions 
such as downed resources, provide specific messages to 
users and IT personnel regarding errors and network 
conditions, and interface with other information 
management, reporting or operational support systems, such 
as policy managers, service level managers, and network 
and system management platforms. Popapps may be cus- 
tomized to add features to existing products, to tailor prod- 
ucts for specific customer needs, and to integrate the 
invented software, systems and methods with technology 
supplied by other vendors. 

Administrator module 134 interacts with various other 
agent modules, maintains and provides network statistics, 
and provides an interface for centrally configuring agents 
and other components of the invented system. With regard 
to agent configuration, administrator module 134 interfaces 
with configuration utility 106 (shown in FIGS. 5 and 13-16), 
in order to configure various agent parameters. Administra- 
tor module 134 also serves as a repository for local reporting 
and statistics information to be communicated to the control 
points. Based on information obtained by other agent 
modules, administrator module 134 maintains local infor- 
mation regarding accessed servers, DNS servers, gateways, 
routers, switches, applications and other resources. This 
information is communicated on request to the control point, 
and may be used for network planning or to dynamically 
alter the behavior of agents. In addition, administrator 
I module 134 stores system policies and/or components of 
I policies, and provides policy data to various agent compo- 
I nents as needed to implement and enforce the policies. 
I Administrator module 134 also includes support for inter- 
/ facing the invented system with standardized network man- 
/ agement protocols and platforms. 

( DNS module 136 provides the agent with configurable 
^-address resolving services. DNS module 136 may include a 
local cache of DNS information, and may be configured to 
first resolve address requests using this local cache. If the 
request cannot be resolved locally, the request is submitted 
to a control point, which resolves the address with its own 
cache, provided the address is in the control point cache and 
the user has permission to access the address. If the request 
cannot be resolved with the control point cache, the con- 
nected control point submits the request to a DNS server for 
resolution. If the address is still not resolved at this point, the 
control point sends a message to the agent, and the agent 
then submits the request directly to its own DNS server for 
resolution. 

DNS module 136 also monitors address requests and 
shares the content of the requests with administrator module 
134. The requests are locally compiled and ultimately pro- 
vided to the control points, which maintain dynamically 
updated lists of the most popular DNS servers. In addition, 
DNS module 136 is adapted to interact with control point 72 
in order to redirect address resolving requests and other 
network requests to alternate targets, if necessary. 

Message broker module 140 creates and maintains con- 
nections to the one or more control points with which the 
agent interacts. The various agent components use the 
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message broker module to communicate with each other and 
with a connected control point. Message broker module 140 
includes message creator and message dispatcher processes 
for creating and sending messages to the control points. The 
message creator process includes member functions, which 
create control point messages by receiving message contents 
as parameters and encoding the contents in a standard 
network format The message creator process also includes 
a member function to decode the messages received from 
the control point and return the contents in a format usable 
by the various components of the agent. 

After encoding by the creator process, control point 
messages are added to a transmission queue and extracted by 
the message dispatcher function for transmission to the 
control point. Messages extracted from the queue are sent to 
the agent's active control point. In addition, the dispatcher 
may be configured to ensure delivery of the message using 
a sequence numbering scheme or other error detection and 
recovery methods. 

Messages and communications from an agent to a control 
point are made using a unicast addressing mechanism. 
Communications from the control point to an agent or agents 
may be made using unicast or a multicast addressing 
scheme. When configured for multicast operation, the con- 
trol point and agents may be set to revert to unicast to allow 
for communication with devices that do not support IP 
multicast. 

Once a connection with a control point is established, 
message broker module 140 monitors the status of the 
connection and switches over to a backup control point upon 
detecting a connection failure. If both the active and backup 
connections are not active, network traffic is passed on 
transparently. 

System services module 142 provides various support 
functions to the other agent components. First, system 
services module maintains dynamic lists of user profiles, 
server profiles, DNS server profiles, control point connec- 
tions and other data. The system services module also 
40 provides a tracing capability for debugging, and timer ser- 
vices for use by other agent components. System services 
module may also be configured with a library of APIs to 
interface the agent with the operating systems and other 
components of the device that the agent is associated with. 

Referring now to FIGS. 10-12, control point 72 and its 
functions will be more particularly described. As seen in 
FIG. 10, control point may include a traffic module 160, a 
server profile module 162, a DNS server profile module 164, 
a gateway profile module 166, an administrator module 168, 
a message broker module 170, a popapp interface 172 and 
a popapp 174. 

Control point traffic module 160 implements policy- 
based, QoS techniques by coordinating the service -level 
enforcement activities of the agents. As part of this function, 
traffic module 160 dynamically allocates bandwidth among 
the agents in its domain by regularly obtaining allocation 
data from the agents, calculating bandwidth allocations for 
each agent based on this data, and communicating the 
calculated allocations to the agents for enforcement. For 
example, control point 72 can be configured to recalculate 
bandwidth allocations every five seconds. During each 
cycle, between re-allocation, the agents restrict bandwidth 
usage by their associated devices to the allocated amount 
and monitor the amount of bandwidth actually used. At the 
end of the cycle, each agent reports the bandwidth usage and 
other allocation data to the control point to be used in 
re-allocating bandwidth. 
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During re-allocation, traffic module 160 divides the total CB, utilized bandwidth UB cannot normally exceed allo- 
bandwidth available for the upcoming cycle among the cated bandwidth AB, because the agent traffic control mod- 
agents within the domain according to the priority data ule enforces the allocation. 

reported by the agents. The result is a configured bandwidth ^ number of processing algorithms may be used to 

CB particular to each mdividual agent, corresponding to that 5 CB.AB and UB for each agent in order to calculate 

agent s fair share of the available bandwidth. The priorities a Qew Mocati however ^ Me ^ eral ^ les 

and configured bandwidths are a function of system policies, ... - . , ~ . °, , r , . ,f, . 

, ~ , , • « . t c \ • 1 j- which are often employed. For example, when bandwidth is 

and may be based on a wide variety of parameters, including , . - j • • c. ? • 1.1 . * . j 

\ «j . j . • • . & taken away from devices, it is often desirable to first reduce 

application identity, user identity, device identity, source „ A . J , , . ' t , , 4 - , , , 4 , 

* *i • » + * j « • r. _ allocations for devices that will be least affected by the 

address, destination address, source port, destination port, inj jj * *tu . a- j i i<-n , 

. i | yn y , . r \ i i j , i 10 downward adjustment. Thus, traffic module 160 may be 

protocol, URL, time of day, network load, network n , 4 J - . , ' - r 4 ' 

. j • * i . . configured to first reduce allocations of clients or other 

populatton, and virtually any other parameter concerning ^ where ^ reports bandwidth usage 

network resources d»t, can be communicated to or ^obtained UB aUocated ^ Presumabl the * e 

by the control pomt. The detail and specificity of client -side , . , t , « . j n_ • n *■ • j j 

J * , r , , J . . . . devices won t be affected if their allocation is reduced, 

parameters mat may oe supplied 10 me control pomt is 15 Generall „ traffic module 160 

not reduce any other 

greatly enhanced in the present invention by the position of n 4 . J ... „ 4 , , „ / r 

& « . , . . . , : r , allocation until all the unused allocations, or portions of 

agent redireaor module 130 relative to the layered commu- ^ ocali have ^ reduced . ^ traffic modme be 

nications protocol stack. The high position within the stack c « ' , „ ... ,. : . 

!• . j , JtL ■ « „ configured to then reduce allocations that are particularly 

allows bandwidth allocation and, more generally, policy ... * m , nr ,- tmanic . „ rA;n „ tn c a rtt L„ - t ^^- 

. A . - j t_ j high, or make adjustments according to some other criteria, 

implementation, to be performed based on very specific ™ 

triggering criteria. This gready enhances the flexibility and Traffic module 160 mav aUo be configured so that when 

power of the invented software, systems and methods. bandwidth becomes available, the newly-available band- 

The priority data reported by the agents may include ™ dth * provisioned according to generalized preferences. 

priority data associated with multiple application programs For f™P k ; ^ |"f c raochllc cai \ bc ^figured l ° P r0Vlde 

• , . j„ • „ f „ ft surplus bandwidth first to agents that have low allocations 

running on a single networked device. In such a situation, f , . * . , , 

the associated agent may be configured to report an "effec- and are requesting additional bandwidOi. After these 

tive application priority," which is a function of the indi- are satlsfied > surplus bandwidth may be apporUoned 

vidual application priorities. For example, if device A were according to priorities or other cntena. 

running two application programs and device B were run- FIGS - 11B > llc and 11D de P lct exam P les of various 

ning a single application program, device A's effective 30 methods that may be implemented by traffic module 160 to 

application priority would be twice that of device B, assum- dynamically allocate bandwidth. FIG. 11A depicts a process 

ing that the individual priorities of all three applications by which traffic module 160 determines whether any adjust- 

were the same. The reported priority data for a device ments to bandwidth allocations AB are necessary. Allocated 

running multiple application programs may be further bandwidths AB for certain agents are adjusted in at least the 

refined by weighting the reported priority based on the 35 following circumstances. First, as seen in steps S4 and SI 0, 

relative degree of activity for each application program. certain allocated bandwidths AB are modified if the sum of 

Thus, in the previous example, if one of the applications all the allocated bandwidths AB total exceeds the sum of the 

running on device A was dormant or idle, the contribution of configured bandwidths CB^,. This situation may occur 

that application to the effective priority of device A would be where, for some reason, a certain portion of the total 

discounted such that, in the end, device A and device B 40 bandwidth available to the agents in a previous cycle 

would have nearly the same effective priority. To determine becomes unavailable, perhaps because it has been reserved 

effective application priority using this weighted method, for another purpose. In such a circumstance, it is important 

the relative degree of activity for an application may be to reduce certain allocations AB to prevent the total alloca- 

measured in terms of bandwidth usage, transmitted packets, lions from exceeding the total bandwidth available during 

or any other activity-indicating criteria. 45 tne upcoming cycle. 

In addition to priority data, each agent may be configured Second, if there are any agents for which AB<CB and 

to report the amount of bandwidth UB used by its associated UB«AB, the allocation for those agents is modified, as seen 

device during the prior period, as discussed above. Data is in steps S6 and S10. The allocations for any such agent are 

also available for each device's allocated bandwidth AB for typically increased. In this situation, an agent has an allo- 

the previous cycle. Traffic module 160 may compare con- 50 cation AB that is less than their configured bandwidth CB, 

figured bandwidth CB, allocated bandwidth AB or utilized i.e. their existing allocation is less than their fair share of the 

bandwidth UB for each device, or any combination of those bandwidth that will be available in the upcoming cycle, 

three parameters to determine the allocations for the upcom- Also, the reported usage UB for the prior cycle is at or near 

ing cycle. To summarize the three parameters, UB is the the enforced allocation AB, and it can thus be assumed that 

amount the networked device used in the prior cycle, AB is 55 more bandwidth would be consumed by the associated 

the maximum amount they were allowed to use, and CB device if its allocation AB were increased, 

specifies the device's "fair share" of available bandwidth for Third, if there are any agents reporting bandwidth usage 

the upcoming cycle. UB that is less than their allocation AB, as determined at step 

Both utilized bandwidth UB and allocated bandwidth AB S8, then the allocation AB for such an agent is reduced for 

may be greater than, equal to, or less than configured 60 the upcoming period to free up the unused bandwidth. Steps 

bandwidth CB. This may happen, for example, when there S4, S6 and S8 may be performed in any suitable order, 

are a number of networked devices using less than their Collectively, these three steps ensure that certain bandwidth 

configured share CB. To efficiently utilize the available allocations are modified, i.e. increased or reduced, if one or 

bandwidth, these unused amounts are allocated to devices more of the following three conditions are true: (1) 

requesting additional bandwidth, with the result being that 65 AB^^CB,^, (2) AB<CB and UBaAB for any agent, or 

some devices are allocated amount AB that exceeds their (3) UB<AB for any agent. If none of these are true, the 

configured fair share CB. Though AB and UB may exceed allocations AB from the prior period are not adjusted. Traffic 
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module 160 modifies allocations AB as necessary at step For example, it may be desired that only a fraction of unused 
S10. After all necessary modifications are made, the control bandwidth be removed during a single reallocation cycle, 
point communicates the new allocations to the agents for Alternatively, the entire unused portion may be removed and 
enforcement during the upcoming cycle. reallocated during the reallocation cycle. 

FIG.11B depicts re-allocation of bandwidth to ensure that s [ n stcp S64 of FIG. 11D, the recovered amounts are 
total allocations AB do not exceed the total bandwidth provisioned as necessary. The recovered bandwidth may be 
available for the upcoming cycle. At step S18, traffic module ^ to cUminate a ^ bctwccn mc total ^0^^ 

160 has determined that the sum of aUocaUo^ ab from the and (he available K bandwidlh) ^ m FIG> 11B) or t0 

prior period exceed the available bandwidth for the upcom- ttuai „ r . . ,,V. . 

in eriod i e AB >CB In this situation certain increase allocations of agents who are requesting additional 

ing peno , i.e. t<xai n * 1 D » ° 10 bandwidth and have relatively low allocations, as in FIG. 

allocations AB must be reduced. As seen in steps S20 and T ..... . ' , , , . ' , 

o-»-i .a= jii^ft u a j.c.j UC. In addition, if there is enough bandwidth recovered, 

S22, traffic module 160 may be configured to first reduce n \ j r . 

' . r . . _i l j_ • ii allocations may be increased for agents requesting addi- 

allocations of agents that report bandwidth usage levels . . . , . im AB & , 1 4 , 6 

, . „ r , * . Im An . *• i Uonal bandwidth, i.e. UBsAB, even where the current 

below their allocated amounts, i.e. UB<AB for a particular n XD - ' . , . c . . ,. . ATJ ™ 

. ™ , ' *■ r *u • allocation AB for such an agent is fairly high, e.g. AB>CB. 

agent These agents are not U smg a porbon of theu 15 ^ ^ ^ meIho(Js fa ^ *g £ ^ 

allocations, and thus are unaffected or only minimally , , , . ... r , „ , , . . - 

„ , \ , . . r ii • recovered bandwidth may be reallocated using a variety of 

affected when the unused portion or the allocation is . _ A r ' „„„ n 

. a a o-irt .l * o- j a i j i methods and according to any suitable criteria, 

removed. At step S20, the traffic module first determines ° ' 

whether there are any such agents. At step S22, the alloca- M indicated, traffic module 160 can be configured to vary 

tions AB for some or all of these agents are reduced. These M & c ratc at wtuch ^ abovc allocation adjustments are made, 
reductions may be gradual, or the entire unused portion of For example, assume that a particular device is allocated 64 
the allocation may be removed at once. ™P S (AB) and rc P orts usa 8 c durirj g thc P rior c y clc of 62 

After any and all unused allocation portions have been ™? s Tr f* C ™ ^ le j 60 cannot ^termine how much 
removed, it is possible that lurther reductions may be addlll0Dal bandwidth the device would use. Thus if the 

required to appropriately reduce the overall allocations M allocation were dramatically increased, say doubled it is 
AB^. As seen in step S24, further reductions are taken P 0ssib ! e * at a S1 g^ cant P™ of "crease would go 
from agents with existing allocations AB that are greater ^ nus ^' However, because the device is using an amount 
than configured bandwidth CB, i.e. AB>CB. In contrast to rou ^ e f al /° J the enforc ^ allocation AB, it can be 
step S22, where allocations were reduced due to unused assumed that the device would use more if the allocation 

bandwidth, bandwidth is removed at step S24 from devices 30 were *«*ased- Thus, it k often preferable to provide small 
with existing allocations that exceed the calculated "fair incremental increases. Hie amount of these incremental 
share- for the upcoming cycle. As seen at step S26, the adjustments and the rate at which they are made may be 
reductions taken at steps S22 and S24 may be performed ^figured with the configuration utility, as will be discussed 
untilthetotalallocationsAB^arelessthanorequaltothe with reference to FIG. 16. If the device consumes the 

total available bandwidth CB W for the upcoming cycle. 3 5 a £* Dm * f ^^ncreases can be provided if 

unj • . *u j r • * *u ii t additional bandwidth is available. 

FIG. 11C depicts a method for increasing the allocation of 
certain agents. As discussed with reference to FIG. 11 A, In addltlon , tne bandwidth allocations and calculations 
where AB<CB and UBsAB for any agent, the allocation AB mav be Performed separately for the transmit and receive 
for such an agent should be increased. The existence of this rates for the networked devices. In other words, the methods 

circumstance has been determined at step S40. To provide *o described ^ reference to FIGS. 11A-11D may be used to 
these agents with additional bandwidth, the allocations for calculate a transmit allocation for a particular device, as well 
certain other agents typically need to be reduced. Similar to as a se P arate receive allocation. Alternatively, the calcula- 
stepsS20 and S22 of FIG. 11B, unutilized bandwidth is first tlons ma y be combined to yield an overall bandwidth 
identified and removed (steps S42 and S44). Again, the allocation. 

control point may be configured to vary the rate at which 45 Server profile module 162, DNS server profile moduleN 
unused allocation portions are removed. If reported data 164, gateway profile module 166 and administrator module J 
does not reflect unutilized bandwidth, traffic module 160 168 all interact with the agents to monitor the status of / 
may be configured to then reduce allocations for agents network resources. More specifically, FIG. 12 provides an / 
having an allocation AB higher than their respective con- illustrative example of how the control points and agents | 

figured share CB, as seen in step S46. The bandwidth 50 may be configured to monitor the status of resources on the \ 
recovered in steps S44 and S46 is then provided to the agents network. The monitored resource(s) may be a server, a DNS 
requesting additional bandwidth. Any number of methods server, a router, gateway, switch, application, etc. At step 
may be used to provision the recovered bandwidth. For S100, a resource status change has occurred. For example, 
example, preference may be given to agents reporting the a server has gone down, traffic through a router has dra- 

largest discrepancy between their allocation AB and their 55 matically increased, a particular application is unavailable, 
configured share CB. Alternatively, preferences may be or the performance of a particular gateway has degraded 
based on application identity, user identity, priority data, beyond a predetermined threshold specified in a system 
other client or system parameters, or any other suitable policy, 
criteria. At step S102, a networked device attempts to access the 

FIG. 11D depicts a general method for reallocating 60 resource or otherwise engages in activity on the network 
unused bandwidth. At step S60, it has been determined that involving the particular resource. If the accessing or request- 
certain allocations AB are not being fully used by the ing device is an agent, an executable spawned by popapp 
respective agents, i.e. UB<AB for at least one agent. At step module 138 (FIG. 9) analyzes the resource, and reports the 
S62, the allocations AB for these agents are reduced. As with identity and status of the resource to the control point 

the reductions and modifications described with reference to 65 connected to the agent, as indicated at step S104. Launch of 
FIGS. 11A, 11B and 11 C, the rate of the adjustment may be the popapp may be triggered by connection errors, or by 
varied through configuration changes to the control point. triggering criteria specified in system policies. For example, 
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system policies may include performance benchmarks for 
various network resources, and may further specify that 
popapp analysis is to be performed when resource perfor- 
mance deviates by a certain amount from the established 
benchmark. In addition, the control points may similarly be 
configured to launch popapps to analyze network resources. 

Once the control point obtains the status information, the 
control point reports the information to all of the agents in 
its domain, and instructs the agents how to handle further 



DNS module 164 also performs certain particularized 
functions in addition to aiding the resource monitoring and 
tracking described with reference to FIG. 12. Specifically, 
the DNS module maintains a local DNS cache for efficient 
local address resolution. As discussed with reference to 
agent DNS module 136, the agents and control points 
interact to resolve address requests, and may be configured 
to resolve addresses by first referencing local DNS data 
maintained by the agents and/or control points. Similar to 



client requests involving the resource, as indicated at steps 10 server profile momjlc 162> DNS module 164 also maintains 



S108 and S110. In the event that the target resource is down, 
underperforming or otherwise unavailable, the instructions 
given to the agents will depend on whether an alternate 
resource is available. The control point stores dynamically 
updated lists of alternate available resources. If an alternate 
resource is available, the instructions provided to the agent 
may include an instruction to transparently redirect the 
request to an alternate resource, as shown in step S108. For 
example, if the control point knows of a server that mirrors 
the data of another server that has gone down, client requests 
to the down server can simply be redirected to the mirror 
server. Alternatively, if no alternate resource is available, the 
agent can be instructed to provide a user message in the 
event of an access attempt, as seen in step S110. The 
messaging function is handled by agent popapp module 138. 
In addition, popapp functionality may be employed by the 
control point to report status information to other control 
points and management platforms supplied by other ven- 
dors. In addition, messages concerning resource status or 
network conditions may be provided via email or paging to 
IT personnel. 

Still referring to FIG. 12, the control point may be 
configured to assume responsibility for tracking the status of 
the resource in order to determine when it again becomes 
available, as shown in step S112. A slow polling technique 
is used to minimize unnecessary traffic on the network. 
During the interval in which the resource is unavailable, the 
agents either redirect requests to the resources or provide 
error messages, based on the instructions provided by the 
control point, as shown in step S116. Once the control point 40 
determines that the resource is again available, the control 
point shares this information with the agents and disables the 
instructions provided in steps S108 and S110, as shown in 
step S118. 

This method of tracking and monitoring resource status 45 
has important advantages. First, it reduces unnecessary and 
frustrating access attempts to unavailable resources. Instead 
of repeatedly attempting to perform a task, a user's requests 
are redirected so that the request can be serviced 
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statistics for use in network planning and dynamic system 
policies. 

In addition to the functions described above, administra- 
tor module 168 maintains control point configuration param- 
eters and distributes this information to the agents within the 
domain. Similar to the server, DNS and gateway modules, 
administrator module 168 also aids in collecting and main- 
taining statistical data concerning network resources. In 
addition, administrator module 168 retrieves policy data 
from centralized policy repositories, and stores the policy 
data locally for use by the control points and agents in 
enforcing system policies. 

Control point 72 also includes a synchronization interface 
(not shown) for synchronizing information among multiple 
control points within the same domain. 

Message broker module 170 performs various functions 
to enable the control point to communicate with the agents. 
Similar to agent message broker module 140, message 
broker module 170 includes message creator and message 
dispatcher processes. The message creator process includes 
member functions that receive message contents as param- 
eters and return encoded messages for transmission to agents 
and other network entities. Functions for decoding received 
messages are also included with the message creator pro- 
cess. The dispatcher process transmits messages and ensures 
reliable delivery through a retry mechanism and error detec- 
tion and recovery methods. 

Referring now to FIGS. 13-16, both the agents and 
control points may be configured using configuration utility 
106. Typically, configuration utility 106 is a platform - 
independent application that provides a graphical user inter- 
face for centrally managing configuration information for 
the control points and agents. To configure the control points 
and agents, the configuration utility interface with adminis- 
trator module 134 of agent 70 and with administrator 
module 168 of control point 72. Alternatively, configuration 
utility 106 may interface with administrator module 168 of 
control point 72, and the control point in turn may interface 



successfully, or the user is provided with information about 50 administrator module 134 of agent 70. 



why the attempts) was unsuccessful. With this information 
in hand, the user is less likely to generate wasteful network 
traffic with repeated access attempts in a short period of 
time. In addition, network traffic is also reduced by having 
only one entity, usually a control point, assume responsibil- 
ity for monitoring a resource that is unavailable. 

In addition to assisting these resource monitoring 
functions, server profile module 162 maintains a dynami- 
cally updated list of the servers accessed by agents within its 
domain. The server statistics may be retrieved using the 60 
configuration utility, or with a variety of other existing 
management platforms. The server statistics may be used for 
network planning, or may be implemented into various 



FIG. 13 depicts a main configuration screen 188 of 
configuration utility 106. As indicated, main configuration 
screen 188 can be used to view various objects managed by 
the invented software, systems and methods, including 
55 users, applications, control points, agents and other network 
entities and resources. For example, screen frame 190 on the 
left side of the main configuration screen 188 may be used 
to present an expandable representation of the control points 
that are configured for the network. 

When a particular control point is selected in the main 
configuration screen 188, various settings for the control 
point may be configured. For example, the name of the 
control point may be edited, agents and other entities may be 
added to the control point's domain, and the control point 



system policies for dynamic enforcement by the agents and 

control points. For example, the control points and agents 65 may be designated as a secondary connection for particular 

can be configured to divert traffic from heavily used servers agents or groups of agents. In addition, the system admin- 

or other resources. istrator may specify the total bandwidth available to agents 
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within the control point's domain for transmitting and 
receiving, as shown in FIG. 14. This bandwidth specification 
will affect the configured bandwidths CB and allocated 
bandwidths AB discussed with reference to control point 
traffic module 160 and the method depicted in FIG. 11. 

Configuration utility 106 also provides for configuration 
of various settings relating to users, applications and 
resources associated with a particular control point. For 
example, users may be grouped together for collective 
treatment, lists of prohibited URLs may be specified for 
particular users or groups of users, and priorities for appli- 
cations may be specified, as shown in FIG. 15. Priorities 
may also be assigned to users or groups of users. As 
discussed above, this priority data plays a role in determin- 
ing bandwidth allocations for the agents and their associated 
devices. 

In addition, optimum and minimum performance levels 
may be established for applications or other tasks using 
network resources. Referring again to the IP telephony 
example discussed above, the configuration utility may be 
used to specify a minimum threshold performance level for 
a networked device running the IP telephony application. 
This performance level may be specified in terms of QoS 
performance parameters such as bandwidth, throughput, 
jitter, delay and loss. The agent module associated with the 
networked device would then monitor the network traffic 
associated with the IP telephony application to ensure that 
performance was above the minimum threshold. If the 
minimum level was not met, the control points and agents 
could interact to reallocate resources and provide the speci- 
fied minimum service level. Similarly, an optimum service 
level may be specified for various network applications and 
tasks. More generally, configuration utility 106 may be 
configured to manage system policies by providing func- 
tionality for authoring, maintaining and storing system 
policies, and for managing retrieval of system policies from 
other locations on a distributed network, such as a dedicated 
policy server. 

Referring now to FIG. 16, the configuration of various 
other control point and agent parameters will be discussed. 
As seen in the figure, configuration utility 106 may be used 
to configure the interval at which resource reallocation is 
performed. For example, the default interval for recalculat- 
ing bandwidth allocations is 5000 milliseconds, or 5 sec- 
onds. Also, as discussed above, the rate at which resource 
allocation occurs may be specified in order to prevent 
overcompensation, unnecessary adjustments to allocations, 
and inefficient reconfigurations. Specifically, the percentage 
of over-utilized bandwidth that is removed from a client 
device and reallocated elsewhere may be specified with the 
configuration utility, as seen in FIG. 16. In addition, the rate 
at which agents provide feedback to the control points 
regarding network conditions or activities of their associated 
devices may be configured. 

It is believed that the disclosure set forth above encom- 
passes multiple distinct inventions with independent utility. 
While each of these inventions has been disclosed in its 
preferred form, the specific embodiments thereof as dis- 
closed and illustrated herein are not to be considered in a 
limiting sense as numerous variations are possible. The 
subject matter of the inventions includes all novel and 
non-obvious combinations and subcombinations of the vari- 
ous elements, features, functions and/or properties disclosed 
herein. No single feature, function, element or property of 
the disclosed embodiments is essential to all of the disclosed 
inventions. Similarly, where the claims recite "a" or "a first" 
element or the equivalent thereof, such claims should be 
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understood to include incorporation of one or more such 
elements, neither requiring nor excluding two or more such 
elements. 

It is believed that the following claims particularly point 
s out certain combinations and subcombinations that are 
directed to one of the disclosed inventions and are novel and 
non-obvious. Inventions embodied in other combinations 
and subcombinations of features, functions, elements and/or 
properties may be claimed through amendment of the 
10 present claims or presentation of new claims in this or a 
related application. Such amended or new claims, whether 
they are directed to a different invention or directed to the 
same invention, whether different, broader, narrower or 
equal in scope to the original claims, are also regarded as 
!5 included within the subject matter of the inventions of the 
present disclosure. 
We claim: 

1. A system for dynamically allocating bandwidth among 
a plurality of distributed client computers interconnected by 

20 a network link, comprising: 

a plurality of agents, each agent being associated with one 

of the plurality of distributed client computers; and 
a control module, wherein 

the agents are adapted to repeatedly communicate 
25 bandwidth allocation data for the distributed client 

computers to the control module; 
in response to the bandwidth allocation data commu- 
nicated by the agents, the control module is adapted 
to dynamically calculate a bandwidth allocation for 
30 each of the distributed client computers and com- 

municate the calculated bandwidth allocation to the 
agent associated with the respective distributed cli- 
ent computer; and 
each agent is adapted to receive the calculated band- 
35 width allocation for its associated distributed client 

computer from the control module and restrict band- 
width usage by the distributed client computer to the 
calculated allocation, the control module being 
adapted to dynamically allocate bandwidth by: 
40 apportioning available bandwidth to determine a 

configured bandwidth share for each distributed 
client computer during an upcoming time period; 
obtaining a prior bandwidth allocation for each dis- 
tributed client computer, wherein the prior band- 
45 width allocation is an amount of bandwidth allot- 

ted to each distributed client computer during a 
prior period; 

obtaining a utilized bandwidth for each distributed 
client computer, wherein the utilized bandwidth is 
50 an amount of bandwidth used by each distributed 

client computer during the prior period; and 
comparing the configured bandwidth share, prior 
bandwidth allocation and utilized bandwidth for 
each distributed client computer to determine an 
55 upcoming bandwidth allocation for each distrib- 

uted client computer for the upcoming period. 

2. The system of claim 1, wherein if the utilized band- 
width is less than the prior bandwidth allocation for any of 
the distributed client computers, the upcoming allocation for 

60 such distributed client computer is determined by reducing 
the prior bandwidth allocation for such distributed client 
computer. 

3. The system of claim 1, wherein if the prior bandwidth 
allocation is less than the configured bandwidth share for 

65 any of the distributed client computers, and if the utilized 
bandwidth is substantially equal to the prior bandwidth 
allocation for any such distributed client computer, the 
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upcoming allocation for such distributed client computer is 
determined by increasing the prior bandwidth allocation for 
such distributed client computer. 

4. A method of dynamically allocating bandwidth to a 
plurality of distributed client computers connected to a s 
network, comprising the steps of: 

periodically obtaining bandwidth allocation data for each 

of the distributed client computers; 
dynamically calculating a bandwidth allocation for each 
of the distributed client computers based on the band- 10 
width allocation data; and 
controlling network access of each of the distributed 
client computers to limit network bandwidth usage of 
each of the distributed client computers to the calcu- s 
lated bandwidth allocations, where such control is 
achieved through operation of an agent software mod- 
ule loaded into memory of each of the distributed client 
computers, wherein the step of dynamically calculating 
a bandwidth allocation for each of the distributed client 
computers is performed by: 
apportioning available bandwidth to determine a con- 
figured bandwidth share for each distributed client 
computer during an upcoming time period; 
obtaining a prior bandwidth allocation for each distrib- 
uted client computer, wherein the prior bandwidth 
allocation is an amount of bandwidth allotted to each 
distributed client computer during a prior period; 
obtaining a utilized bandwidth for each distributed 
client computer, wherein utilized bandwidth is an 3Q 
amount of bandwidth used by each distributed client 
computer during the prior period; and 
comparing the configured bandwidth share, prior band- 
width allocation and utilized bandwidth for each 
distributed client computer to determine a bandwidth 35 
allocation for each distributed client computer for the 
upcoming period. 

5. The method of claim 4, wherein if the utilized band- 
width is less than the prior bandwidth allocation for any of 
the distributed client computers, the upcoming allocation for 4Q 
such distributed client computer is determined by reducing 
the prior bandwidth allocation for such distributed client 
computer. 

6. The method of claim 4, wherein if the prior bandwidth 
allocation is less than the configured bandwidth share for 45 
any of the distributed client computers, and if the utilized 
bandwidth is substantially equal to the prior bandwidth 



allocation for any such distributed client computer, the 
upcoming allocation for such distributed client computer is 
determined by increasing the prior bandwidth allocation for 
such distributed client computer. 

7. A system for dynamically allocating bandwidth among 
a plurality of distributed client computers interconnected by 
a network link, comprising: 

a plurality of agents, each agent being associated with one 
of the plurality of distributed client computers and 
adapted to monitor and control its associated distrib- 
uted client computer by interfacing with a socket object 
used to operatively couple the network link with appli- 
cation programs running on the associated distributed 
client computer; and 
a control module, wherein: 

the agents are adapted to communicate bandwidth 
usage data for the distributed client computers to the 
control module; 
the control module is adapted to dynamically calculate 
a bandwidth allocation for each of the agents in 
response to the bandwidth usage data; and 
each agent is adapted to interface with the socket object 
actively restrict bandwidth usage on the network link 
by its associated distributed client computer in accor- 
dance with the bandwidth allocation calculated for 
that agent, wherein the control module is adapted to 
dynamically allocate bandwidth by: 
apportioning available bandwidth to determine a 
configured bandwidth share for each distributed 
client computer during an upcoming time period; 
obtaining a prior bandwidth allocation for each dis- 
tributed client computer, wherein the prior band- 
width allocation is an amount of bandwidth allot- 
ted to each distributed client computer during a 
prior period; 

obtaining a utilized bandwidth for each distributed 
client computer, wherein the utilized bandwidth is 
an amount of bandwidth used by each computer 
during the prior period; and 

comparing the configured bandwidth share, prior 
bandwidth allocation and utilized bandwidth for 
each distributed client computer to determine an 
upcoming bandwidth allocation for each distrib- 
uted client computer for the upcoming period. 
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